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The minimization of Gibbs free energy is based on the changes in work and free energy that occur 
in a physical or chemical system. The maximization of mutual information, the capacity, of a noisy 
channel is determined based on the marginal probabilities and conditional entropies associated with a 
communications system. As different as the procedures might first appear, through the exploration 
of a simple, "dual use" Ising model, it is seen that the two concepts are in fact the same. In 
particular, the case of a binary symmetric channel is calculated in detail. 



INTRODUCTION 

In 1876 J.W. Gibbs [l[ proposed a refinement 
to the notion of Helmholtz free energy. The util- 
ity of the refinement rests on the observation 
that when attempting to determine the max- 
imum amount of work that may be extracted 
from a thermodynamic process, some of that 
work may already be accounted for, for exam- 
ple, work against the atmosphere —pAV. The 
suggestion was that perhaps the actual quan- 
tity of interest is the maximum amount of work 
other than atmospheric that may be extracted. 
This general concept, equipped with a system 
dependent form for the work term [H , [H , [3] , [B| is 
well known and widely used in physics, physical 
chemistry and many engineering disciplines. 

Some seventy years later C. Shannon and W. 
Weaver Q provide a sound theoretical frame- 
work for determining the maximum amount 
of information that may be transmitted from 
sender to receiver through a noisy channel. 
Their concept, the channel capacity, calculated 
by extremization of differences of various com- 
munication system entropies, is reminiscent of 
statistical physics Q but apparently different 

The purpose of this letter is to show how these 
two concepts are in fact the same. To accom- 
plish this it is useful to have a simple example 
that belongs to both the physics and communi- 
cations theory traditions. The Ising model for 
magnetic systems is a good candidate. The liter- 
ature on the topic is vast. Physical applications 
of the model relevant to the present purpose in- 



clude [9| , [l(| , U | • Relevant applications of the 
model in communications include 12\. 13|.[l4j|. 
After a brief review of the Ising model, it will be 
shown that the real space renormalization pro- 
cedure [15| connects the Gibbs free energy and 
the channel capacity in a natural way. 



REVIEW 

As noted in the introduction, the Ising model 
is of interest in both physics and communica- 
tions. The states described by the model are a 
sequence of electron spin up (f) or spin down 
Q) states which are easily translated to the l's 
and 0's of a bit stream. The simplest communi- 
cations scheme is one in which the transmitted 
bits are statistically independent of each other 
but coupled to the bits received. The sequence 
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FIG. 1: A simple send/receive Ising model. The sent 
bits, <7i, 03, <T3 (odd numbered) are independent of 
each other but coupled to their received bits 02 , 04 , 
<76 (even numbered). 
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shown in figure [T] consists of six sites represent- 
ing 2 6 possible configurations of spins (bits). 

Following the paradigm of Boltzmann and 
Gibbs, the probability weight of each spin con- 
figuration is a function of its own particular en- 
ergy as well as the energy of its environment, 
represented by the temperature 17 1. Al- 



scnd 



though at first the pedagogy may seem spe- 
cialized to a narrow class of systems occurring 
in physics, engineering and chemistry, the ap- 
proach is quite general and any discrete prob- 
ability distribution {pi ,P2,- ■ ■ ,vn } can be put 
into Boltzmann-Gibbs form [181 ]. 

The assumed statistical independence of the 
send/receive pairs manifests itself in terms of the 
physics through the form of the energy function. 
Under these conditions, the energy of a config- 
uration, i?cri,...,tr6) is comprised of additive con- 
tributions from the distinct send/receive pairs 
with no interaction energy terms 
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FIG. 2: The possible states of the basic unit (one 
send, one receive) and corresponding numerical val- 
ues. 



-"trl,...,cr6 — H ai fj 2 -\- H a3 aA -f- H a5 fj G . 

For this reason, to understand the properties 
of the total system, it is enough to study the 
properties of a single send/receive pair. A typ- 
ical Ising Hamiltonian, appropriate for a single 
send/receive unit, for example site 1 (send) and 
site 2 (receive) as shown in figure [TJ is of the 
form 

H ai a 2 = -Jo-i<7 2 + h(ai + er 2 ) + c 

with spin coupling J and external field h. The 
so-called Zeeman energy term, c, is initially zero. 
Later, as a result of the decimation, the con- 
straints imposed by the channel and energy min- 
imization, it will be assigned a value. 

The four possible states of a basic unit are 
listed in figure O For a concrete example, 
is computed as 

— J(l * -1) + h(l + -1) = J. 

The remaining energies are computed similarly. 

The following conventions and definitions are 
standard and will be referred to in the sequel. 
Let (3, defined as the inverse of the send/receive 



unit system temperature, be constant (isother- 
mal) . The Hclmholtz free energy of the system, 
is defined by 

Z = 

where the partition function (the sum over all 
configurations) is given by 

Z = e -^TT + e -0Hn + e -0Hn + e -/JH u . 

The probability of any particular spin configura- 
tion, for example tt (send a "1", receive a "1") 
is represented in Boltzmann Gibbs form as 



The concept of internal energy, denoted U, is 
a macroscopic observable and a common object 
of study in thermodynamics. Changes in the in- 
ternal energy , At/, are decomposed into two 
distinct macroscopic types: work AW and heat 
AQ. As noted in the introduction, macroscopic 
descriptions of work are varied and system de- 
pendent. Thermodynamic expressions of Gibbs 
free energy often inherit this system dependent 
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language. Statistical mechanics offers an addi- 
tional point of view. In terms of the microscopic 
data, the internal energy associated with a ba- 
sic send /receive unit is also well represented by 



K 



17] 



Hi 



or equivalently 



-Log(P m ) + H r[ 



u — Pffff|f + Pn^n + ^Vr-^i-T + Pa^ii- 

Similarly, the generic form of work in terms of 
given changes in the microscopic data is [l8| 



AW = P TT AP| T + P n AH n -{ 
P U AH U + P U AH U . 



ff T = -Log(P in ) 



H 



II- 



In the parlance of communications theory, the 
"sender's hamiltonian" , H, determines (via 
Boltzmann Gibbs) the "sender's marginal" , P. 
Notice that the channel is invariant under a 
transformation 



A TRIVIAL DECIMATION 

A channel is represented mathematically as a 
matrix of conditional probabilities. For exam- 
ple, the probability that the receiver observes a 
| given that a | was sent: Pjjf. The channel for 
the send/receive unit system introduced in the 
last section is simply 



P 



lit 



TIT 

P Tli P ili 

where the rows of the matrix sum to one. In 
terms of the Boltzmann Gibbs distribution, the 
probability of receiving a spin up given that a 
spin up was sent is 



p _ c 

T 1 T e -0H n + e -Wn ■ 



The probability of receiving a spin down given 
that a spin up was sent is 



ilT 



t + e 



-/3H r 



Define the standard [l5| rescaled Hamiltonian, 
H, by summing out (decimation) the spin values 
at site 2 (the "receiver") 



e -m, =e -f3H n+e -0H U 



so that by the definition of the channel 



— ► ^*TT C T 

P| — > P| + C-f . 

In a similar way, and cj are obtained in terms 
of the original unit system and channel via dec- 
imation. 

A single constant is sufficient to arbitrarily 
adjust the sender's marginal at fixed channel. 
Without loss of generality set 



-c T . 



Interestingly, from a pedagogical point of view, 
this is the only choice consistent with the as- 
sumption of a canonically distributed unit sys- 
tem (l9j . The renormalized (sender's) partition 
function, Z , is defined 



Z = e-^r 



Some Quantities of Physical Interest 

The capacity is defined as the (maximized) 
difference of two information theoretic quanti- 
ties: the entropy associated with the receiver's 
marginal distribution, S{P) and the average of 
the channel entropies according to the sender, 
< S 'channel >p- But what is the connection to 
the physics? 

Consider first < S channel >p- This is sim- 
ply (— (5 times) the work done on the four state 
system by the H decimation. For example 

P T (-P m Log(P m ) - P m Log(P m )) 



4 



is easily seen to be 

with the Pj terms making a similar contribution. 

Secondly, the receiver's entropy, S(P) is cal- 
culated in terms of the sender's marginal P and 
the channel columns as 





■PLogf 


P TIT 
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P ilT 


■pLogf 
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Exercise the right to decimate the system over 
the spin at site 1 and define H 



-0Hf 



-f3H n 



-0H n 



The entropy of the receiver's marginal may then 
be expressed 

/3(P TT i7 T + P iT i? T + P n H l + P ll H l + j 3 Lo 9 Z). 

Putting the pieces together, it is seen that the 
channel capacity, i.e. the (maximized) quantity 

S{P) < Schannel >p 

is given by the maximum over Cf of 

P »([H. + H.] - H..) + LogZ. (1) 

spins 

This is easily seen to be (—(3 times) the change 
in the Gibbs free energy 



-AW + AT 



where the change in work, — AW (for a fa- 
mous, macroscopic example, — (— pdV) ) and the 
change in Hclmholtz free energy, AT, are asso- 
ciated with separating a composite microscopic 
spin configuration, for example f j, into its com- 
ponent spins, t and |, via decimations. By in- 
spection, the work term, AW, in equation |T]) is 
apparent in its microscopic, statistical mechan- 
ical form "pAif" . A moments reflection identi- 
fies the free energy changes. For by the defini- 
tion of the decimated partition functions Z and 



Z, the change in the Helmholtz free energy, AT, 
is given by 

-^([LogZ + LogZ] - LogZ) = - ^LogZ. 

In this way it is seen that determining the ca- 
pacity of the two site spin system is equivalent 
to the standard Gibbs free energy analysis. The 
capacity is the maximum amount of work (other 
than the work done by the sender/receiver deci- 
mations) that may be extracted from the system 
consistent with the channel constraints. 



Symmetric Channel 

The prototypical example of a simple commu- 
nications system is the binary symmetric chan- 
nel. In the context of the Ising prototype dis- 
cussed in this letter, the channel symmetry is 
achieved by setting the applied field h equal to 
zero. The hamiltonian 



depends on the coupling term J and 



where 



P TIT P iW 
P \\i P i\i 



5 = 



1-5 5 
5 1-5 



-PJ 



-0j 



As described in the previous section, decimating 
over the "receivers spin" defines the H hamilto- 
nian 



H } = -±Log(e 
Hi = -jjLog (e" 



-pj + e /3J 



)+c T 
c T . 



Recall that the inclusion of cj leaves the channel 
invariant and is sufficient to arbitrarily adjust P, 
the senders marginal distribution. 

Obtain H similarly, i.e. by decimation over 
the "senders spin" and respecting the channel 
constraints 

# T = _ l Lo g ( e -0(J-°T) + e W- C T)) 
H l = -j. Log ( e -/J(J+c T ) + e 0(J+c r )) . 
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With these quantities in hand, the Gibbs free 
energy changes consistent with the channel con- 
straints may be computed, following equation 
(TTJ) , as a function of c j . Figure [3] shows the re- 
sults for the case (3 = 1, J = 1. With these 
parameter settings and accompanying channel 
constraints, the minimum AG, i.e. —1 times 
the channel capacity, is achieved at c-f = 0. 
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SUMMARY 
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